Real-time Reinforcement Learning Control of Dynamic Systems Applied to an Inverted Pendulum

This paper describes work which was started in order to investigate the use of neural networks for application in adaptive or learning control systems. Neural networks have learning capabilities and they can be used to realize non-linear mappings. These are attractive features which could make them useful building blocks for non-linear adaptive or learning controllers. The work can be motivated as follows. For some processes, only ill-defined models are available to the control engineer. Despite this problem, these processes need to be controlled in some way. Neural networks may be useful here because they are able to learn such a model without a preliminary choice of the model structure. After learning the model structure is represented by the structure and (non linear) elements of the neural network. The model parameters are represented by the weight factors of the network. Until now, only well known processes have been used in neural control research. The reason for this is that much of the behavior of neural networks is unknown. Convergence and stability of network learning algorithms have not been proved. The simple problems regarded here are needed to study learning behavior and to interpret the knowledge obtained by learning. This is necessary in order to estimate the complexity of problems which can be solved using neural control. One of the main problems in the neural control field is that neural networks have never been designed or regarded from this point of view. The concepts of neural networks have been developed by scientists from various disciplines. Among them were biologists, physisists, psychologists, mathematicians and so on. After the f i t succesful applications, (control) engineers became interested in neural networks. The historical development of neural network research has resulted in a number of algorithms which have often been developed from a biological point of view. These algorithms are now studied by engineers to investigate their possible use in various technological fields. The work described in this paper concems such an algorithm. The first part of the paper will be concemed with a short introduction. The concepts of adaptive control theory are outlinedand some neural network paradigms will be described which could be useful for control purposes. The field of neural networks is regarded through the eyes of a control engineer. It will be shown that neural networks can be seen as complex multivariable adaptive systems. Simple neural networks can be configured in such a way that adaptive controller configurations appear which are well known. Furthermore the similarities in adaptation rules of adaptive controllers and neural networks are shown. Finally the direct and indirect adaptive control schemes will be introduced in which the neural network paradigms of reinforcement learning and supervised learning (using back propagation) may be used. The second part of the paper consists of a case study about a reinforcement learning control system published by Barto et a1 (1983) and worked out by Anderson (1987). Although not designed for this purpose, it is used for the control of physical dynamical systems. it will be shown that this algorithm can be regarded as a direct adaptive control scheme. The meaning of the algonthm will be reviewed in a control engineering context revealing its attractive and less attractive features. It will be shown that the algorithm is not generally applicable to control problems. However, some minor modifications solve this problem.